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ABSTRACT 

We present deep iJ5T'/^CS' observations in Si475r625«7752850 towards the z — 4.1 radio galaxy TN 
J1338-1942 and its overdensity of >30 spectroscopically confirmed Lya emitters (LAEs). We select 
66 (?475-band dropouts to zsso.So- — 27, 6 of which are also a LAE. Although our color-color selection 
results in a relatively broad redshift range centered on z = 4.1, the field of TN J1338-1942 is richer 
than the average field at the >5cr significance, based on a comparison with GOODS. The angular 
distribution is filamentary with about half of the objects clustered near the radio galaxy, and a small, 
excess signal {2a) in the projected pair counts at separations oi 9 < 10" is interpreted as being 
due to physical pairs. The LAEs are young (a few xlO'^ yr), small {{rui) = 0'.'13) galaxies, and 
we derive a mean stellar mass of ^ 10^~^ M0 based on a stacked iiTg-band image. We determine 
star formation rates, sizes, morphologies, and color-magnitude relations of the (7475-dropouts and find 
no evidence for a difference between galaxies near TN J1338-1942 and in the field. We conclude 
that environmental trends as observed in clusters at much lower redshift are either not yet present, 
or are washed out by the relatively broad selection in redshift. The large galaxy overdensity, its 
corresponding mass overdensity and the sub-clustering at the approximate redshift of TN J 1338-1 942 
suggest the assemblage of a > 10^^ M0 structure, confirming that it is possible to find and study 
cluster progenitors in the linear regime at z > 4. 

Subject headings: cosmology: observations - early universe - large-scale structure of universe - galax- 
ies: high-redshift - galaxies: clusters: general - galaxies: starburst - galaxies: 
individual (TN J1338--1942) 



1. INTRODUCTION 

The formation and evolution of structure in the Universe 
is a fundamental field of research in cosmology. Clus- 
ters of galaxies represent the most extreme deviation 
from initial conditions in the Universe, and are there- 
fore good evolutionary probes for studying the formation 
of the large-scale structure. While clusters of galaxies 
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have been studied extensively in the relatively nearby 
Universe, their evolutionary histor y becomes obscure be- 
yond rough ly half the H ubble tim e ("Blakes lee et al.|[2006l : 
iMullis et a l.' 2005; Stanf ord et al . 2006). Their progeni- 
tors are extremely difficult to identify when the density 
contrast between the forming cluster and the field be- 
comes small, and mass condensations o n the scales of 
clusters are extremely rare at any epoch (lKaiseiiri984[ ). 

Even the most distant clusters known contain, be- 
sides a population of star-forming galaxies, an older 
population of relat i vely red and massive galaxies (e.g. 
Dressier et aIlll999l:Fran Dokkum et~al1l2000l: iGoto et al.l 



20051) . The scatter in the color-magnitude relation for 
cluster ellipticals at z 1 is virtually indistinguishable 
from that at low redshift, suggesting that some of the 
galaxy populations in cl usters have very old stellar pop- 
ulations (e.g . [Stanford e t al.lll998|: iBlakeslee et al.ll2003l : 
iWuvts et al.1 12004. : ,Holden et al.l I2OO50 . The epoch of 
cluster and cluster galaxy formation is presumed to be 
marked by the violent build-up of the stellar mass and 
morphologies of these early-type galaxies. 

Large samples of star-forming Lyman break galaxies 
(LBGs) that are selected in the rest-frame UV have 
been used to probe the cosmic star formation rate his- 
tory as well as the large-scale structure far beyond 
z = 1. The epoch corresponding to the peak in 
the star formation rate density (z ~ 2 — 3) is pre- 
ceded by a significant but modest decrease in the star 
formation rate density from z ^ 3 out to z ^ 6 
(Madau et al.' 1996; Steidel et all 119991 : iGiavalisco et all 
,2004a : Bouwens et al. 2007). Deep near-IR NICMOS ob- 
servations in and around the GOODS fields find small 
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samp l es of z ^ ^ ~.§ galax ies ([Bouwens fc lUingworthl 
120061 iBouwens etldl [20043) and indicate that this de- 
cUne in the nu mber of UV luminous galaxies continues 
from 2; ^ 7 — 8 (jBouwens fc Illingwor th 2006) . This sug- 
gests that cosmic star formation is a rather extended 
process, with only modest but continual changes over 
time. 

LBGs, as well as the partially overlapping popula- 
tion of Lya emitters (LAEs), are strongly clustered at 
z = 3 — 5, and are highly biased relative to predictions 
for the d ark mat ter dist ribution dGiavalisco et al.|[l998t 
Adelberg eT et al.l Il998l: lOuchi et all l2004bl ; iLee et all 
2006t iKashikawa et al.l [200l) . The biasing becomes 
stronger f or galaxies with higher rest -fr ame UV lu- 
minosity ("Giavalis co fc DickinsonI |2001D . lOuchi et al.l 
([2004b) found that the bias may also increase with red- 
shift and dust extinction. They suggest that the red- 
dest LBGs could be connected wit h sub-mm sources 
or the extremely red objects (EROs, lElston et al.|[T988l : 
IMcCarthv et"al]l200ll : [Paddi et al.l[200l ). By comparing 
the numb er densities of LBGs to that of dark halos pre- 
dicted bv lSheth fc TormenI (|1999[ ). they concluded that 
LBGs at z ~ 4 are hosted by halos of 1 x 10" - 5 x lO^^ 
M0, and that the descendants of those halos at z = 
have masses that are comparable to the masses of groups 
and clusters. The derived halo occupation numbers of 
LBGs increase with luminosity from a few tenths to 
roughly unity, implying that there is only one-to-one 
correspondence between halos and LBGs at the highest 
masses. 

Studies of sizes indicate that high-redshift star- forming 
galaxies are compact in size 0.1"-0.3"), while 

large {> OA") l ow sur face brightness galaxies are rare 
(jBouwens et al.l l2004al) . and there is a clear decrease 
in size with redshift for objects of fixed luminosity 
([Ferguson et al.ll2004t [Bouwens et a"ni2004aD . Morpho- 
logical analysis of LBGs indicates that they often possess 
brighter nuclei and more disturbed profiles than local 
Hubble types de graded to the same image quality (e.g. 
iLotz et al.(l2"00l . 

Despite these advances in the study of the evolution of 
the highest redshift galaxies, galaxy clusters have been 
studied out to only z = 1.5 while it is clear that they be- 
gan forming at much earlier epochs. Finding and study- 
ing these cluster progenitors may yield powerful tests 
for (semi-)analytical models and A'^-body simulations of 
structure formation, and give new clues to how the most 
massive structures in the Universe came about. 

Several good candidates for galaxy overdensities, pos- 
sibly 'protoclust ers^*^', have bee n discover ed at very 
high redshift (e.g. iPascarelle et al.|[l9 "96: Keel e t al lll999l: 



Francis et al."2001; Moller & Fynbo 2001; Steidel et al.' 
199a. .2005..; Shimasaku et al. 2003 ; Ouchi et al., 2005b ; 
Kashikawa et al. I 120071) . These structures have been 



found often as by-products of wide field surveys using 
broad or narrow band imaging. 

Luminous radio galaxies have also been used to search 
for overdense regions within the large-scale structure 
at high redshift. The association of distant, power- 

^'^ The term protocluster is commonly used to describe non- 
virialized galaxy overdensities at high redshift (z > 2) that, when 
evolved to the present day, have estimated masses comparable to 
those of the virialized galaxy clusters. See Sect. 6 for a more 
physical definition. 



ful radio galaxies with massive galaxy and cluster for- 
mation is mainly based on two observational clues. 
First, some high red shift radio galaxies are very mas- 
sive s vstems^^ Ce.g. [ De Breuck et al.l l200d iDey et alj 



y et ai.l 

.1 I2005t 

Milev et al. 2006: Scvmour et al. .2007[ ), suggesting that 



1997t IPentcricci et al.' '2001': 'Villar-Martm et al 



their host galaxies are likely to be the progenitors of 
massive red sequence galaxies or even the brightest 
cluster galaxies (BCGs) that dominate the deep po- 
tential wells of clusters. Second, in some cases high 
redshift radio galaxies have been shown to have com- 
panion galaxies. At 1.5 < z < 2, there is evidence 
for overdensities of red galaxies associated with ra- 
dio sources, c onsistent with moderately rich Abe l l-type 
clusters (e.g. [S anchez fc Gonzalez-Serrano 1999, 200^; 
Thompson et al. 2000: HaU et al. 2001; Best et al. 200j; 
Wold et al.ll200l . At z > 2, several radio galaxies have 



been found to be surrounded by overdensities of LAEs 
of a few, discovered through deep narrow band imaging 
and spectroscopic follow-up with the Very Large Tele- 
scope ( VLT) of the Euro pean Southern Observatory (e.g. 
Pentericci ct al. 2000; Kurk et al.ll2003l : IVenemans et al.l 



2002112004 ,2005 . ,2007,) . 



Focussing on the excesses of LAEs discovered in the 
vicinity of distant radio sources, we are performing a 
survey of LBGs in such radio-selected protoclusters with 
the Advanced Camera for Surveys on t he Hubble Space 
Telesc ope fHST/ACS: iFord et al.l [l99l. In iMilev et all 
([20041 ) and I Overzier et alT i 2006aD we reported on the de- 
tection of a significant population of LBGs in the (pro- 
jected) region around radio galaxies at z — 4.1 (TN 
J1338-1942) and z = 5.2 (TN J0924-2201). Here, we 
will present a detailed analysis of the ACS observations 
of protocluster TN J1338-1942 at z = 4.1, augmented 
by ground-based observations with the VLT. This struc- 
ture is amongst the handful of overdense regions so far 
discovered at z > 4, as evidenced by 37 LAEs that repre- 
sent a surface overdens i ty of ^ 5 compared to other fields 
(IVenemans et al.l [20021 [2001 . The FWHM of the veloc- 
ity distribution of the LAEs is 625 km s~^. The mass 
overdensity as well as the velocity structure is consistent 
with the global proper ties of z ^ 4 protoclu s ters derived 
from simulations (e.g. iDe Lucia et al.l [20041 : iSuwa et all 
120061) . The radio galaxy itself is highly luminous in the 
rest-frame UV, optical and the sub-mm, indicating a high 
star formation rate. It has a complex morphology which 
we have interpreted as arising from AGN feedback on 
the forming ISM and a massive starburst-driven wind 
(jZirm et al.l[2005[) . 

The main issues that we will attempt to address here 
are the following. What are the star formation histories, 
physical sizes and morphologies of LAEs and LBGs? In 
particular we wish to study these properties in relation to 
the overdense environment that the TN J1338-1942 field 
is believed to be associated with, analogous to galaxy 
environmental dependencies obs erved at lower redshifts 
and predicted by models (e.g. [ Kauffma nn et al.l 12004 
iPostman et"all l2005t IDe Lucia et al. 200i). How does 
sub-clustering associated with the TN J1338-1942 struc- 

We note that the measure of the absolute stellar masses 
of galaxies is still a matter of debate given the different imple- 
mentations in the literature of the post-AGB stellar phase in the 
modeling of the rest f rame near-infrared galaxy emission (e.g., see 
IMaraston eral|[2003: IBruzual|[2007l: lEminian et~al|[2007l) . 
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ture compare to the 'field' ? What is the mass overdensity 
of the structure, and what is the relation to lower redshift 
galaxy clusters? In Sect. 2 we will describe the obser- 
vations, data reduction and methods. We present our 
sample of LBGs in Sect. 3, and describe the rest-frame 
UV and optical properties of LBGs and LAEs. In Sect. 4 
we present the results of a nonparametric morphological 
analysis. In Sect. 5 we will present further evidence for a 
galaxy overdensity associated with TN J1338-1942 and 
investigate its clustering properties. We conclude with a 
summary and discussion of our main results (Sect. 6). 
We use a cosmology in which gp = 72 km s~^ M pc~^, 
VIm = 0.27, and f^A = 0.73 (jSpergel et al.l[2003h . The 
luminosity distance is 37.1 Gpc and the angular scale 
size is 6.9 kpc arcsec^^ a.t z = 4.1. The lookback time is 
11.9 Gyr, corresponding to an epoch when the Universe 
was approximately 11% of its current age. All colors and 
magnit udes quoted in this paper are expressed in the AB 
system (|Okel ll97H ). 

2. OBSERVATIONS AND DATA REDUCTION 

2.1. ACS imaging 

We observed one field with the ACS around the radio 
galaxy TN J1338-1942 (henceforward 'TN1338'). These 
observations were part of the ACS Guaranteed Time Ob- 
serving high redshift cluster program. To search for 
candidate cluster members on the basis of a Lyman- 
break at the approximate wavelength of Lya redshifted 
to z = 4.1, we used the Wide Field Channel to obtain 
imaging through the broadband filters^^ 5475, ?'625, ^775, 
and zgso (Fig. [T]). The total observing time of 18 orbits 
was split into 9400 s in each of (7475 and rQ25, and 11700 
s in each of 1775 and zsso- 

Each orbit of observation time was split into two 1200 
s exposures to facilitate the removal of cosmic rays. The 
data were reduced using the ACS pipeline science in- 
vestigation software {Apsis; Blakcslec et al. 2003). Af- 
ter initial processing of the raw data through CALACS 
at STScI (bias/dark subtraction and flat-fielding), the 
following processing was performed by Apsis: empiri- 
cal determination of image offsets and rotations using 
a triangle matching algorithm, background subtraction, 
the rejection of cosmic rays and the geometric correc- 
tion and combining of exposures through drizzling using 
the STSDAS Dither package. The final science images 
have a scale of 0'.'05 pixel"^. The total field of view is 
11.7 arcmin^. The radio galaxy (aj2ooo = 13''38™30^ 
<5j20oo = — 19°42'30") is located about 1' away from 
the image centre. The field further in cludes 12 of the 
37 spectros copically confirmed LAEs (jVenemans et al.l 
|2002[ I2007t l. The resultant color image of the field is 
shown in Fig. [21 with 3475 in blue, re25 in green, and zsso 
in red. The radio galaxy clearly stands out as the sole 
'green' object in the entire field, due to it s prominent halo 
of Lyg emission obser ved in rQ25 (see IVenemans et al.] 
[200a iZirm erani2005h . ^ 

We used the ACS zeropoints from ISirianni et al.l 



()2005), and an extin c tion value of E{B — V) = 0.096 mag 
from ISchlegel et al.l ()1998i) . We measured the limiting 
magnitudes from the RMS of noise fluctuations in 10 000 

^® We use 3475, r625, 2775 and zgrM to denote magnitudes in 
the HST/ACS passbands F475W, F625W, F775W and F850LP, 
respectively, or to denote the passbands themselves. 



square apertures of varying size that were distributed 
over the images in regions free of objects. Details of the 
observations are given in Tabled] 

2.2. VLT optical spectroscopy and NIR imaging 

We obtained 10 hours of VLT/F0RS2 spectroscopy in 
service modc"'^^. The instrumental setup, the seeing con- 
ditions, and the meth od of processing of the data were 
similar as described in IVenemans et al] (|2002l . [2b05( ). 

Near-infrared data in the -ftTg-band were obtained with 
VLT/ISAAC^^ We observed a 2'Ax2'A field for 2.1 hours 
in March 2002, and for 5.4 hours in a partly overlapping 
field in 2004. After dark subtraction, flat fielding and 
rejection of science frames of poor quality, the data for 
each night was individually processed into a combined 
image using the XDIMSUM package in IRAF. Since only 
the data taken on the night of March 26 2002 was con- 
sidered photometric, the combined images of the other 
nights were scaled to match that particular night using 
several unsaturated stars for reference. We derived the 
zeropoint based on observations of the near-IR photo- 
metric standard FS 142. However, we had to adjust the 
zeropoint by 0.2 magnitudes to match the magnitudes of 
several 2MASS stars in the field. The seeing was ~0'.'5 
(FWHM), and the galactic extinction in Ks was 0.036 
mag. 

Next, each of the combined images with the native 
ISAAC scale of 0'.'148 pixel"^ was corrected for geometric 
distortion by projecting it onto the 0'.'05 pixel^^ ACS 1775 
image using the tasks GEDMAP/GEDTRAN in IRAF. The 
projection had a typical accuracy of ~1.5 ACS pixels 
(RMS). The registered images were combined using a 
weighting based on the variance measured in a source- 
free region of each image. The limiting 2a depth in the 
AB^^ system was 25.2 magnitudes for a circular aperture 
of 1'.'4 diameter. Areas that are only covered by either 
the 2002 or the 2004 data are shallower by 0.5 and 0.3 
mag, respectively. The i^g-band data cover 81% of the 
ACS field, and contain the radio galaxy and 11 LAEs. 

2.3. Object detection and photometry 

Object detection and photometry was done using 
the SE xtractor software package of iBertin fc ArnoutsI 
(|1996l ). We used SExtractor in double -image mode, 
where object detection and aperture determination are 
carried out on the so-called "detection image" , and the 
photometry is carried out on the individual filter im- 
ages. For the detection image we used an inverse vari- 
ance weighted average of the r625, ^775 and zgso images, 
and a map of the total exposure time per pixel was used 
as the detection weight map. Photometric errors were 
calculated using the root mean square (RMS) images 
from Apsis. These images contain the absolute error per 
pixel for each output science image. We detected ob- 
jects by requiring a minimum of 5 connected pixels at 
a threshold of 1.5 times the local background {S/N of 

17 Program ID: 071.A-0495(A) 

18 Program ID: 073.A-0286(A) 

1* IRAF is distributed by the National Optical Astronomy Ob- 
servatories, which are operated by the Association of Universities 
for Research in Astronomy, Inc., under cooperative agreement with 
the National Science Foundation. 

^ s ,Vega 
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> 3.35). The values for SExtractor's deblending param- 
eters (DEBLEND_MINCONT = 0.1, DEBLENDJfTHRESH = 8) 
were chosen to hmit the extent to which our often clumpy 
z ^ A (7475-dropouts were split into multiple objects. Our 
'raw' detection catalog contained 3994 objects. We re- 
jected all objects which had S/N less than 5 in 2:350 , 
where we define S/N as the ratio of counts in the isopho- 
tal aperture to the errors on the counts. The remaining 
2022 objects were considered real objects, although they 
still contain a small fraction of artefacts and objects that 
were split up. 

We used SExtractor's MAG_AUTD to estimate total mag- 
nitud es within an aperture radius of 2.5 x TKron ()Kronl 
|198C1() . However, when accurate color estimation is more 
important than estimating a galaxy's total flux, for ex- 
ample in the case of color-selection or when determin- 
ing photometric redshifts, isophotal magnitudes are pre- 
ferred because of the higher S/N and the smaller con- 
tribution of neighboring sources. Therefore we calculate 
galaxy colors from isophotal magnitudes. These proce- 
dures are optimal for (faint) object detection a nd aper- 
ture photometry with ACS (jBeiiiftez et al.ll2004D . 

Optical- NIR (observed-frame) colors were derived from 
combining the ACS data with lower-resolution ground- 
based Ks data in the following way. We used PSFMATCH 
in IRAF to determine the 2D kernel that matches the 
point spread function (PSF) in the ACS images to that 
obtained in the Ks-hand, and convolved the ACS images 
with this kernel. The photometry was done using SEx- 
tractor in double image mode, using the Ks-hand image 
for object detection. Colors involving the NIR data were 
determined in circular apertures with a diameter of 1'.'4, 
although we used a 3'.'0 diameter aperture for the large 
radio galaxy. 

2.4. Aperture and completeness corrections 

The photometric properties of galaxies are usually mea- 
sured using source extraction algorithms such as SEx- 
tractor. We can conveniently use this software to deter- 
mine aperture corrections and completeness limits as a 
function of e.g., the 'intrinsic' or real apparent magni- 
tude, half-light radius {rhi) or the shape of the galax; 



surface brightness profi le (see also iBenitez et al.l I20& 
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iGiavalisco et al.ir2004bf l. To this end we populated the 
ACS zgso image with artificial galaxies consisting of a 
50/50 mix of exponential and de Vaucouleurs profiles. 
We simulated ~ 10 000 galaxies with ~ 200 per simu- 
lated image to avoid over-crowding. We took uniformly 
distributed half-light radii in the range O'.'l-l'.'O, and 
uniformly distributed axial ratios in the range 0.1-1.0. 
Galaxies were placed on the images with random posi- 
tion angles on the sky. Using the zeropoint we scaled the 
counts of each galaxy to uniformly populate the range 
2850= 20 — 28 mag. We added Poisson noise to the simu- 
lated profiles, and convolved with the zgso PSF. SExtrac- 
tor was used to recover the model galaxies as described 
in Sect. [231 

Approximately 75% of the artificial galaxies were de- 
tected. In Fig. [3] (left panel) we show the measured 
rhi versus the input 'intrinsic' r^i- Radii are increas- 
ingly underestimated as the input radii become larger, 
because the surface brightness gets fainter as r^; . On av- 
erage, the radius is underestimated by about 50% for a 
0850"^ 26 mag object with an intrinsic half-light radius of 



0'.'4. The discrepancy between input and output radius 
is generally smaller for an exponential than for a de Vau- 
couleurs profile. In Fig. [3] (middle panel) we show the 
aperture corrections defined by the difference between 
MAG-AUTO and the total simulated magnitude. The 
amount of flux missed rises significantly towards fainter 
magnitudes, with a 0.5-1.0 mag correction for objects 
with output magnitudes of Z850 = 25 — 27 mag. Finally, in 
Fig. [3] (right panel) we show the z^so completeness limits 
as a function of zg5o and r^i ■ About 50% completeness is 
reached at ^850= 26 — 26.5 mag for unresolved or slightly 
resolved sources. Note that the 50% completeness limit 
will lie at measured MAG_AUTO magnitudes that are 
fainter by 0.5-1.0 mag, given the significant aperture cor- 
rections shown in the middle panel of Fig. [3l 

Throughout the paper, we apply approximate correc- 
tions to the physical quantities derived from measured 
rhi and magnitudes (e.g. physical sizes, luminosities, and 
SFRs) based on the above results for exponential profiles. 
Angular sizes and magnitudes quoted are always as mea- 
sured. 

2.5. Photometric redshift technique 

W e used the Bay esian Photometric Redshift code (BPZ) 
of lBenite3 ()2000[ ) to estimate galaxy redshifts, zb- For 
a complete description of BPZ an d the robustne ss of 
its results, w e refer the reader to iBenitezj ()2000D and 
iBem'tez et al.l (|2004l ). Our library of galaxy spectra is 
based on the elliptical, intermediate (Sbc ) and late type 
spiral {Scd), and irregular templates of IColeman et all 
(|1980t ). augmented by two starburst galaxy templates 
with E{B - y) ^ 0.3 (SB2) and E{B - F) ~ 0.45 
{SB3) from lKinnev et all (|1996f) . and two simple stellar 
popu lation (SSP) models with ages of 5 Myr a nd 25 Myr 
fromlBruzual k Chariot (|2003l ). As shown bv lCoe et all 
(|2006| ). the latter two templates improve the accuracy of 
BPZ for very blue, young high r edshift galaxies in th e 
Hubble Ultra Deep Field (UDF, iBeckwith et"aI1l2006f) . 
BPZ uses a parameter 'ODDS' defined as P{\z — zb\ < 
Az) that gives the total probability that the true red- 
shift is within an uncertainty Az. For the uncertainty we 
can take the empirical accuracy of BPZ for the HDF-N 
which has a = 0.06(1 -I- zb)- For a Gaussian probabil- 
ity distribution a 2a confidence interval centered on 
would get an ODDS of > 0.95. The empirical accuracy 
of BPZ is (T « 0.1(1 + Zb) for objects with /814 < 24 and 
z < 4 observed in the i3435V606^8i4-bands with ACS to 
a dep th comparable to our observations (|Benitez et al.l 
I2004f ). Note that we will be applying BPZ to generally 
fainter objects at z ~ 4 observed in (?475r625i775Z85o. The 
true accuracy for such a sample has yet to be determined 
empirically. The accuracy of BPZ may be improved by 
using certain priors. We apply the commonly used mag- 
nitude prior that is based on the magnitude distribution 
of galaxies in real observations (e.g. the HDF). 

2.6. Template-based color-color selection of protocluster 
LBG candidates 

We extracted LBGs from our catalogs using color crite- 
ria that are optimized for de t ecting star-forming galax- 
ies at z ^ 4 ( St eide l_^LalJ 119991 : lOuchi et al.ri2004al : 
IGiavalisco et al.ll2004af ). To define the optimal selection 
for our c hoice of filters we fo llowed the approach em- 
ployed by iMadau et al.l ()1996[ ). We used the evolution- 
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ary stellar population synthesis model code GALAXEV 
(Bruzual & Chariot 2003) to simulate a large variety of 
galaxy spectral energy distributions (SEDs) using: (i) 
the Padoya 19 94 simple stellar population model with a 
ISalpeteJ ([1955) initial mass function with lower and up- 
per mass cutoffs = 0.1 Mq and mu = 100 Mq of 
three metallicities (0.2^0, 0.4^0, Z©), and (ii) the pre- 
defined star formation histories for instantaneous burst, 
exponentially declining (r = 0.01 Gyr) and constant 
{t — 0.1, 1.0 Gyr) star formation. We extracted spec- 
tra with ages bet ween 1 Myr and 13 Gyr, applied the 
reddening law of ICalzetti et all ()2000f ) with E{B - V) 
of 0.0-0.5 mag, and redshifted each spectrum to red- 
shifts between 0.001 and 6.0, including the effects of at- 
tenuation by the in tergalactic medium (IGM) using the 
iMadau et a'fl (|1996i ) recipe. Galaxies were required to be 
younger than the age of the Universe at their redshift, 
but other parameters were not tied to redshift. While 
this approach is rather simplistic due to the fact that the 
model spectra are not directly tied to real observed spec- 
tra and luminosity functions, it is reasonable to expect 
that they at least span the range of allowed physical spec- 
tra. The resulting library can then be used to define a set 
of color criteria for selecting star-forming galaxies at the 
appropriate redshift, and estimating color-completeness 
and contamination (Madau et al. 1996). 

We extracted the model colors by folding each spec- 
trum through the corresponding ACS filter transmission 
curves. No photometric scatter was applied to the mod- 
els. The 3475-^625 and 7-625-2:850 color-color diagram used 
to isolate LBGs at z ~ 4 is shown in Fig. [H We elected 
to use the r625-Z85o color in defining o ur selection region 
(instead of the r625-«775 color used in lMilev et al.l[2003 ) 
due to the greater leverage in wavelength. 

The color-color region that we use to select z ^ 4.1 
LBGs is defined as: 



ff475 - rQ25 > 1-5, 
ff475 - '"625 > ''625 - ^850 
''625 - ^850 < 1-0. 



1.1, 



(1) 

Fig. [5] shows the color selection efficiency as a function 
of redshift, defined as the number of galaxies selected 
in a redshift bin, divided by the total number of model 
galaxies in that redshift bin. The dotted histogram indi- 
cates the fraction of model galaxies meeting the selection 
criteria. The resulting redshift distribution has an ap- 
proximately constant maximum efficiency of ~ 30% for 
3.5 < z < 4.5. If we limit the model galaxies to ages less 
than 100 Myr and < E{B - V") < 0.3 (consistent with 
the average LBG populat ion at z 3 — 4 ()Papovich et al.l 
120011: ISteidel et al.lll999f l). the color completeness (sohd 
histogram) becomes ^ 90% for models at z ^ 4.1. 
The dashed histogram shows the fraction of models with 
ages greater than 0.5 Gyr selected, illustrating the main 
sources of contamination in our z ^ 4.1 sample, namely 
from relatively old galaxies at z ~ 2.5 and the possible 
inclusion of Balmer-break objects at z ^ 0.5. 

2.7. GOODS simulated images 
2.7.1. The simulations 

To determine whether TNI 338 is also host to an over- 
density of LBGs at z ~ 4.1, we will want to compare 
the number of (7475-dropouts found in our ACS field with 



that found in a random field on the sky. Unfortunately, 
at present, there are not many ACS fields available, with 
comparable depths in 5475, r625, and Z85o to carry out 
such comparison. We therefore avail ourselves of the 
four-band GOODS field to make these comparisons. The 
3 orbit i?435, 2.5 orbit Vgoej 2.5 orbit 1775, and 5 orbit 
2850 coverage is similar in depth and much larger in cover- 
age, to the g475?'625*775285o imaging we have on TN1338, 
suggesting that with simple wavelength interpolation, we 
should be able to mirror our TN1338 selection. 

Though there are many ways to have performed this 
interpolation, we chose to perform the interpolation di- 
rectly on the ACS data itself, changing it from the ob- 
served S435V606«7752850 filter Set tO the 3475 ''625 « 775 2850 

filter set. This transformation was performed on a pixel- 
by-pixel basis, using the formula: 

fL=I^.j g{SED,Y,irn,z) + 



( \\iY)-\{X)\ 
^^=^- \\{Xh)^\{Xl) 
g{SED,Y,X,z)Af^„ 



(2) 



where f^j is the flux at pixel («, j) in some band Y, lij is 
the best-fit flux in each pixel (expressed as an Z775-band 
flux), g{SED,Y, X, z) is a generalized fc-correction from 
some band X to another band Y for some SED and 
redshift z, and X{X) is the mean wavelength for some 

band X. The summation S^^j^^ runs over those bands 
which immediately straddle the Y band, and the A/^^ 
terms account for the error in the fits to individual pixels. 
The best-fit fluxes lij were determined by minimizing 



IiJg{SED,X,i775,z)-f- 



(3) 



where //^- and afj are the flux and its uncertainty, re- 
spectively, in the X band at pixel position The 
error terms A/,^ are equal to fi,j — g{SED,X,iTr5, z). 
The first term in Eq. [2] is a generalized fc-correction ap- 
plied to the best-fit model SEDs, while the second is an 
interpolation applied to the fiux residuals from the fit. 
T his is nearly identical to expressions from Appendix Bl 
of iBouwens et al.l ()2003af l and represents a slight update 
to that procedure. 

The redshift, z, and spectral energy distribution, SED, 
that we use for individual pixels are based upon an ini- 
tial object catalog we made of each field before doing 
the transformation Objects are detected off a image 
(jSzalav et al.l Il999f ) constructed from the V606*7752850- 
band using a fairly aggressive 3ct threshold and splitting 
parameter (SExtractor DEBLEND_MINCONT=0.005). 
Best-fit redshifts and SEDs are then estimated for each 
object from the photometry. These model parameters, 
in turn, are assigned to all the pixels which make up 
these objects (according to the SExtractor deblending 
maps), and thus used in the transformation given by 
Eq.O Only objects having colors of (-B435 — Vgoo) > 0.8, 

(^435 - V^606) > 0.6(^606 " 2350) + 0.5, (B435 " Vgoo) > 

3.375(V(306 — 285o) — 4.575 were transformed. These colour 
limits were derived from the standard i3435-dropout se- 
lection region (see Fig. 8 in Bouwens et al. 2007), but 
are less restrictive in order to account for the fact that 
objects which drop out of the (7475-band should have a 
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slightly higher mean redshift than objects which drop 
out of the i3435-band. Therefore, we need to expand the 
size of our selection window to select galaxies at slightly 
higher mean redshifts on average, i.e. by taking galax- 
ies with a redder VeoQ-zg^Q color. Because objects with 
higher redshift will also drop out of the Vgoe-band sooner, 
we have to decrease the colour cut in -B435-V6O6 as well 
because the limit we can set on the -B435-V6O6 colours 
will be weaker. The selection region thus includes all 
objects expected in a standard _B435-dropout selection, 
but was expanded to allow for objects at slightly higher 
and lower redshifts that are potentially important when 
transforming the images to the TN1338 filter set. 

Since our ACS reduction of the TN1338 field had a 
different pixel scale ( i.e., 0.^05) than that of the GOODS 
vl.O reduction (0"03: lGiavalisco et al.l[2004bf ). we did not 
use that reduction as the basis for our simulation of the 
CDF-S GOODS field. Instead, we made use of an in- 
dependent reduction we had made of the GOODS field 
with Apsis. That reduction was performed on a 0'.'05 
gr id, using a procedure n early identical to that described 
in iBouwens et al.l (|2006D . but using a 'Lanzcos3' kernel 
(which matches the TN1338 ACS reduction). 

2.7.2. ReUabtltty of the simulations 

The image transformation method described in the pre- 
vious section provides us with a convenient way of con- 
verting the i?435V606i775Z85oiiiiagiiig data available in the 
GOODS field into the 5475^625 1775 ^sso bands used for 
the TN1338 g-dropout selection. In the limit of infinite 
S/N data set and perfect model SEDs, the results ob- 
tained from this transformation should be close to per- 
fect. However, in the real world with finite S/N, it be- 
comes necessary for us to test this method to see how 
well it works in practice. We do this by generating two 
different simulations of the same field and then repeat- 
ing the same g475-dropout selection on each simulation. 
In the first simulation, we generate the 5475 r625i 7752^850 
image set directly from some input model, and in the 
second simulation, we first use this same model to gener- 
ate a i?435V606*775'Z850 image set (to mimic the GOODS 
data in both the depth and passband coverage) and then 
convert this image set to the 5475 ?'625«775 2^850 bands using 
the image transformation method described in the pre- 
vious section. By comparing the f;475-dropout selections 
obtained from both methods, we can examine the effect 
that this transformation method has on our g-dropout se- 
lection. The same input catalog is used for both simular - 
tions and was generated using the IBouwens et al.l ()2007D 
z ~ 4 LP and a mean J/y-continuum slope of —1.5 with 
Icr scatter of 0.6. The profiles for s ources in this c atalog 
were taken from real sources in the IBouwens et all (j2007l ) 
HUDF i3-dropout sample. The redshift of the input ob- 
jects for the simulation ranged from z = 3 to z = 5 and 
the limiting magnitude was ^850= 27.5. 

Objects were extracted from the two mock TN1338 
datasets as described in Sect. 12.31 and we selected a 
(7475-dropout sample from each of the two datasets using 
the selection criteria detailed in Sect. 12.61 In Fig. El 
we compare the color-color diagram of objects detected 
in both the direct simulations and those obtained from 
our image transformation method, respectively. Objects 
(not) qualifying as 5475-dropouts as defined in Sect. 12.61 
are indicated by (small) large symbols. In order to make 



a fair comparison, we have excluded objects at z < 3.5 
and z > 4.5 in the left panel, because such objects are 
not present in the right panel due to the wide B435- 
dropout color selection that is applied when transform- 
ing the GOODS images into the TN1338 images (see 
Sect. I2.7.ip . Note that the transformation from model 
to GOODS to TN1338 {right panel) introduces some 
small changes in the color distributions with respect to 
the transformation directly from model to TN1338 {left 
panel). This is due to the fact that in the right panel 
object detection and color selection are performed twice, 
and because of uncertainties introduced by determining 
the photometric redshifts of the objects in the simulated 
GOODS images. However, using different limiting zgso 
magnitudes of [26.5,26.0,25.5] the ratio of the number 
of g475-dropouts detected/selected in our direct simula- 
tions to that in our transformed GOODS images was 
[0.84,1.17,1.13]. At the faint end, our image transforma- 
tion method (comparable to the method used for trans- 
forming the real GOODS dataset into a TN1338 dataset) 
thus slightly overproduces the actual number of 5475- 
dropouts expected based on the UDF input model. At 
the brighter limiting magnitudes, our image transforma- 
tion method slightly underproduces the number expected 
from the direct method (at 2;85o< 25.5 the difference of 
13% represents a difference of just one object). Overall, 
we conclude that the (7475-dropout selections performed 
on the transformed images to be fairly similar (within 
~ 20%) in general in terms of the overall numbers to that 
found on data obtained directly in those bands. Thus, 
we are confident in comparing the (7475-dropout counts 
found in the transformed GOODS images to that found 
in the real TN1338 data (see Sect. [5]). 

3. RESULTS 

In this section we apply the color-color section to the 
TN1338 field to select a sample of candidate z ^ 4.1 
LBGs ((7475-dropouts) and study their properties in Sect. 
13.11 In Sect. 13.21 we will study the same properties for 
the sample of 2: = 4.1 LAEs. 

3.1. The gij^-dropout sample 

Using the selection criteria defined in Eq. [T]we extracted 
a sample of g f475-dropouts. Although the stellar locus 
(|Pickleslll998li lies outside the region defined by our se- 
lection criteria, we additionally required objects to have a 
SExtractor stellarity index of < 0.85 (non-stellar objects 
with high confidence). Our final sample consisted of 66 
objects with Z85o<27.0 mag, 51 of which have Z850<26.5 
mag, and 32 of which have 2;850<26.0 mag. The color- 
color diagram is shown in Fig. [T] 

3.1.1. Star formation rates 

The characteristic luminosity, L*=4, of the LBG lumi- 
nosity function at z ^ A corresponds to -Z850~ 25.0 
(jSteidel et al.lll999l ). The sample contains two objects, 
one of which is the radio galaxy, with a luminosity of 
^ 6L* (1775 ~ 23). The remainder of the sample spans 
luminosities in the range 0.4 — 2L*, where we have ap- 
plied aperture corrections of up to ~ 1 magnitude based 
on the exponential profiles in Fig. |3l 

We calculated star formation rates (SFRs) from the 
emission-line free UV flux at 1500 A (1775) using the con- 
version between luminosity and SFR for a Salpeter initial 
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mass function (IMF) given in lMadau etall (|1998f ): SFR 

(Mo yr-i) = L^^^^^ (erg s"! Hz-i)/8 x lO^^. For ages 

that are larger than the average time that late-O/early- 
B stars spend on the main sequence, the UV luminosity 
is proportional to the SFR, relatively independent of the 
prior star formation history. The SFRs are listed in Ta- 
ble [3] The radio galaxy and object #367 each have a 
SFR of ^ 95 Mq yr^i. The median SFR of the entire 
sample is ~ 8 Mq yr~^. Although we assumed here that 
the LBGs are dust-free, one could multiply the SFRs by 
a factor of 2.5 to correct for an average LEG extinction of 
E{B — V) » 0.1 mag (see next section) giving a median 
SFR of - 20 Mq yr-i. 

3.1.2. UV Continuum colors 

We calculate the UV continuum slopes from the 1775-2850 
color. This color spans the rest-frame wavelength range 
from ~ 1400 A to - 2000 A. We assume a standard 
power-law spectrum with slope f3 {fx oc A''', so that a 
spectrum that is flat in has /3 = —2). We calculate 



ft; 



logi 



?775 



0.4(^775 



ZS50) 



(4) 



where A775 and A850 are the effective bandpass wave- 
lengths, and Q775 and Qsso are the fractions of the 
continuum fluxes remaining after applying the recipe 
for fo reground neutral hydrogen absorption of iMadaul 
(|1995f ). The break at rest-frame 1216 A only starts to 
enter the Z775-band for galaxies at z > 4.7. Thus Q775 
and Qs5o are unity and (3 will be relatively independent of 
redshift for 3.5 < z < 4.5. The measured slopes are plot- 
ted in Fig. [H Excluding the two brightest sources, we 
find (ft;: ) = —1.95. This is sig nificantly bluer than that 
found bv lPapovich et ahl ()200ir ). although it is consistent 
at the bright magnitude end where the comparison with 
L* galaxies is appropriate (thin dashed line). 

We have modeled the dependencies of the slope on 
age and dust using an exponential star formation his- 
tory (r = 10 Myr) with Q.2Zq metallicity and a Salpeter 
IMF. For a constant E{B — V)~ 0.0 the range of slopes 
favours ages in the range 50-300 Myr. Although the 
incompleteness for faint, relatively red objects could b e 
higher than for blue objects (e.g. lOuchi et al.l l2004al ). 
a high dust content {E{B — V) ~ 0.3) is incompatible 
with the majority of the slopes observed. A linear fit 
to the data gives a slope-magnitude relation of ftz = 
(-0.16 ± 0.05)(z85o - 25 mag) - 1.84 ± 0.11 (thick solid 
line), which remains virtually unchanged when we ex- 
clude the two brightest objects (thick dashed fine). Our 
relation is in good agreement with that of -B435-dropouts 
in GOODS (R.J. Bouwens, private communication). The 
best-fit relation spans ages in the range 5-150 Myr for a 
constant E{B — V)k, 0.1. A similar slope-magnitude re- 
lation is also observ ed in other works (|Meurer et al .11 19991 : 
lOuchi et al.ll2004aD and may imply a mass-extinction or 
a mass-metallicity relation rather than a relation with 
age. Interpreting the slope-magnitude relation as a mass- 
extinction relation implies E{B — V) k, 0.13 mag at 
Z850~ 23 mag and E{B — V) k, 0.0 mag at zgsow 27 
mag for a fixed age of 70 Myr. 

3.1.3. Rest-frame UV to optical colors 



At 2; 4.1, the filters 1775, 2850 and Ks probe the rest- 
frame at ~1500, 1800, and 4300A, respectively. We de- 
tected 13 of the 5475-dropouts in the iiTs-band at > 2(7. 
In Fig. [9] we show the irr^-Ks versus 1775-^850 color di- 
agram. The in^-Ks color is more sensitive to the effects 
of age and dust than 1775-2850 j due to its longer lever arm 
in wavelength. Comparing the colors to the best-fit LBG 
spectrum from lPapovich et al.l ()2001[ ) redshifted to 2 ~ 4 
shows that the observed colors are consistent with ages 
in the range 10-100 Myr, although there will be degener- 
acy with dust. Non-detections in the i^s-band suggests 
that more than 50% of the g475-dropouts have ages less 
than 70 Myr, with a significant fraction less than 30 Myr. 
The radio galaxy is among the reddest objects, although 
it has large gradients in irr^ -Ks among its va rious stellar 
and AGN components (see iZirm et aLll2005D . 

In Fig. [TO] we pl ot the irr^^-Ks versus Ks color- 
magnitude diagram. iPapovich et"all (|2004D found evi- 
dence for a trend of generally redder colors for galaxies 
that are brighter in Ks in GOODS. The effect is not 
likely to be a sel ection effect because t he objects are se- 
lected in the UV. iPapovich et al.l ()2004D suggest that age 
or dust of LBGs at 2 ~ 3 — 4 may increase with increas- 
ing rest-frame optical luminosity. Our data are consistent 
with that conclusion. 

A comparison with an exponentially de c lining star- 
burst model track from iBruzual fc CharlotI (|2003( ) indi- 
cates that the stellar masses of the objects in our sample 
span about one order of magnitude, ranging from ^ 10^ 
Mq for the faintest LBGs to 10^*' M© for the brightest. 

3.1.4. Sizes 

We measured the half-light radius, r/j/, in 2350 using 
SExtractor by analysing the growth curve for each ob- 
ject out to 2.5 X TKron- Excluding the exceptionally large 
radio galaxy, the measured radii range from unresolved 
(^ 0'.'07) to 0'.'42, corresponding to physical diameters of 
< 7 kpc at 2 ~ 4. The average radius is 0'.'17 or ~ 1.4 
kpc. If we divide our sample into two magnitude bins 
each containing an approximately equal number of ob- 
jects (achieved by placing a cut at 285o=26.1 magnitude), 
the mean rhi are 0'.'21 ± O'.'Ol (error represents the stan- 
dard deviation of the mean) and 0'.'14±0'.'01 in the bright 
and faint bins, respectively. The difference is expected to 
be largely due to a larger flux loss in the fainter sample 
(see Fig. [3]), although fainter galaxies could indeed be 
smaller because of the r2oo ~ ~ L^^^ luminosity-size 
relationship, where r2oo is the virial radi us and Vc is t he 
circular dark matter halo velocity (see , Mo et al.lfT998f l. 

The i775-band morphologies of the g475-dropouts are 
shown in Fig. [TT] A separate section (Sect. [4|) is devoted 
to a non-parametric morphological analysis of our sample 
and a comparable field sample. 

3.1.5. Point sources 

The requirement for objects to be non-stellar to make it 
into our sample had no big effect on the selection, since 
only 4 sources that passed our color-color selection crite- 
ria were rejected based on high stellarity index. However, 
one of the point sources corresponds to a spectroscopi- 
cally confirmed Lya emitter (see next section) , indicating 
that at least one of the point sources is a genuine z — A.l 
LBG. We derive an upper limit for the fraction of 2 ^ 4 
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sources missed that are unresolved (intrinsically small or 
AGN) of -6%. 

3.2. Lya galaxies 
IVenemans et al.l ()2002l |20Q7[ ) found an overdensity of 
37 LAEs {EWo^Lya > ISA), all spcctroscopically con- 
firmed to lie within 625 ±150 km s^^ of the radio galaxy 
z = 4.11. All of the 12 LAEs in the ACS field have been 
detected in rQ25, 1775 and zgao (see Tabled. Their pr op- 
erties are described below (see also lMilev et al.l[2004l ). 

3.2.1. Star formation rates 

The Z850 magnitudes are in the range 25.3-27.4 mag, cor- 
responding to a luminosity range of ~ 0.2 — l.OL*. The 
SFRs are ~ 1 - 14 Mq yr~^ with a m edian of 5.1 M(r, 
yr~^ ( not including the effect of dust). IVenemans et al.l 
(|2005D calculated the SFRs from Lya using SFR (Up, 
YlZll= 8.7LLya (erg s-i)/1.12 x lO^'^, from iKennicutti 
()1998[) with the standard assumption of case B recom- 
bination ([Brocklehurst 1971, Lua/ Lhya = 8.7 for gas 
that is optically thick to HI resonance scattering and no 
dust). In general, we find good agreement between the 
SFRs calculated from the UV compared to Lya with a 
median UV-to-Lya SFR ratio of 1.3. 

3.2.2. UV continuum colors 

The UV slopes are indicated in Fig. [5] (stars). The slope 
can be constrained relatively well for the four brightest 
emitters, which have -2.1 ± 0.4, -2.0 ± 0.6, -1.9 ± 0.7, 
and —2.5 ± 0.5. The LAE slopes scatter around the piz- 
magnitude relation for the (/475-dropouts found in Sect. 
I3.1.2[ with a sample average of — 1.7± 1.2. These slopes 
are consistent with a flat (in f^) continuum, thereby 
favouring relatively low ages and little dust. 

3.2.3. Rest-frame UV to optical colors 

None of the 1 1 LAEs covered by the Ks image were de- 
tected at the > 2(7 level. We created a stack of the Ks- 
band fluxes for the 5 LAEs that lie in the deepest part 
of our NIR image. The subsample had 25.3 <2;850< 26.4 
mag and ((1775-2850)) ~ 0.0. We obtained a 3cr detec- 
tion for the stack finding Ks = 25.81q32 mag and hence 
iTjr^-KgKi 0.0. We compared this to 100 stacks of 5 5475- 
dropouts each, selected from a sample of 12 that had a 
similar range in Z85o-band magnitudes and j775-2;85o col- 
ors as the LAEs. The average detection among the 100 
stacks was ~5ct, corresponding to Ks = 25.13lQ^g mag 
and i775-Ks^ 0.7. The results from the stacks have been 
indicated in Figs. [9] and [TOl Although the difference in 
the i775-Ks color is significant at only ~ 2(7, one inter- 
pretation is that the faintest LAEs have slightly lower 
stellar masses (a few times 10* Mq) compared to LBGs 
(see Sect. |6]). 

3.2.4. Sizes and morphologies 

If the Lya emission is associated with an extended halo 
of outflowing gas, we might find differences in the typical 
radii of the sources measured in the filter that includes 
Lya compared to pure continuum filters. We calculated 
rfii^r from the r625-band, the filter that includes Lya, 
and compared it to the Vhi.z of the continuum calculated 
from the 2;85o-band (Table [2]). The mean r^i are 0'.'13 in 



rg25 and 0'.'12 in Z850- z — 4.1, the measured angular 
sizes correspond to physical radii of < 3 kpc, with a 
mean value of ~ 1 kpc. We do not find evidence for 
the sources to be more extended in rQ25 than they are in 
Z850, suggesting that Lya emission either coincides with 
the continuum region, or originates from a very extended, 
low surface brightness halo not detected with ACS. One 
exception is source L7 which has r^i r = O'.'IS compared 
to rhl,^ = 0'.'13 and rhi,z = O'.'ll. 

We have measured the rhi from a sample of 17 field 
stars in a similar magnitude range. The stars were se- 
lected on the basis of SExtractor stellarity index of 1.0. 
Four of the LAEs (L4, Lll, L20, L22) have a rui in both 
bands that is indistinguishable from that of the stars. 
The UV luminosities of these unresolved LAEs are no 
different than those of the resolved ones. Spectra show 
that the Lya lines are narrow (<1000 km s~^) and no 
high ionization lines have been detected fiVenemans et al.l 
l2002l ). ruling out that they are broad emission line AGN. 
The light is probably due to unresolved stellar regions 
with rhi < 500 pc. If we restrict ourselves to resolved 
sources only, the mean rhi are 0'.'15 in both r625 and 

Z850- 

The ACS morphologies at rest-frame ~ 1500A are 
shown in Fig. [121 Two sources (L16 & L25) have double 
nuclei separated by ^ 0'.'5 3 kpc) that are connected 
by faint, diffuse emission suggestive of merging systems. 

4. MORPHOLOGICAL ANALYSIS 

In order to quantify the wide range in ACS continuum 
morphologies seen in Fig. IIH we have carried out a non- 
parametric morph ological an a lysis of the (7475-dropout 
sample. Following iLotz et ahl (|2004j) we determined the 
following morphological coefficients: 1) The Gini coeffi- 
cient (G), a statistic for the relative distribution of an 
object's flux over its associated pixels, 2) Af2o, the nor- 
malised second order moment of the brightest 20% of a 
galaxy's pixels, and 3) concentration (C), the ratio of the 
circ ular radii conta ining 20% and 80% of the total flux 
(see IConselicel[2003l and references therein). 

To improve the S/N of our sample we coadded the 1775 
and Z850 images, giving a total exposure time of 23500 s. 
To further improve the S/N per pixel the images were 
binned using a 2 x 2 binning scheme. Pixels were flagged 
as belonging to an object if they were inside one 'Pet- 
rosian radius' (Pctrosian 1976). This ensures that the 
morphological analysis is relatively insensitive to vary- 
ing surf ace brightness li mits and S/N among different 
objects ()Lotz et al.ll2004D . The number of (7475-dropouts 
with sufficient {S / N)pixei was maximized by setting the 
free Petrosian parameter to 0.3. We measured the mor- 
phologies for a total of 15 of the (7475-dropouts that had 
{S/N)pixei > 3, which included the radio galaxy. The co- 
efficients were all determined within a maximum radius 
of 1.5 X Tp. We used SExtractor's segmentation maps to 
mask out all pixels suspected of belonging to unrelated 
sources. Errors on the coefficients were determined us- 
ing Monte Carlo simulations. The value of each pixel 
was modified in such a way that the distribution of val- 
ues were normally distributed with a standard deviation 
equal to that given by the RMS image value for the cor- 
responding pixel. We remark that the concentration (C) 
is often an underestimate of the true concentration of 
high redshift objects, because the 20% flux radius is often 
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smaller than the ~ 0.1" resolution obtained by HST . 

In Fig. [13] we plot the parameters measured for the 
TN1338 (jf475-dropouts (blue points). The radio galaxy 
(large circle) has non-average values in each of the pa- 
rameter spaces, owi ng to its co mplex morphology as de- 
scribed in detail bv IZirm et al.l ([2005.). We have applied 
a similar morphological analysis to a sample of 70 LBGs 
at z 4 with {S/N)pixei > 3 selected from the CDF- 
S GOODS field for comparison. Fig. [13] indicates that 
both the centroids and the spread of the TN1338 pa- 
rameter distributions (blue contours) coincide with that 
of the parameter distributions determined from GOODS 
(red contours). 

5. EVIDENCE FOR AN OVERDENSITY ASSOCIATED 
WITH TN J1338-1942 AT Z = 4.1? 

5.1. Surface density distribution 

In Fig. [14] we show the angular distribution of the 3475- 
dropouts in TN1338. Contours of the local object surface 
density compared to the average field surface density il- 
lustrate that the 5475-dropouts lie predominantly in a 
filamentary structure. Another density enhancement is 
located near the top edge of the image. Interestingly, 
both peaks in the object surface density distribution co- 
incide with an extremely bright ~ 6L* LBG. One of these 
is the radio galaxy, which takes a central position in the 
largest concentration of (7475-dropouts in the field. About 
half of the dropouts lies in a 2.'1 x 2'.1 region that includes 
the radio galaxy. 

The (7475-dropouts detected in Ks are found only in 
the high density regions, most notably in the clump to 
the left of the radio galaxy. The filamentary distribution 
of the dropouts is not seen in the spatial distribution of 
the 12 LAEs, which are distributed more uniformly over 
the field. In fact, 8 of the emitters lie in regions that 
are underdense compared to the overall distribution of 
(7475-dropouts. 

5.2. Comparison with 'field' LBGs from GOODS 

Here we will test whether the structure of 5475-dropouts 
represents an overdensity of star-forming galaxies asso- 
ciated with TN J1338 -1942, similar to the overdensity 
of LAEs discovered bv lVenemans et al.l (|2002, 2007). To 
determine the 'field' surface density of (7475-dropouts we 
have extracted a control sample by applying our selection 
criteria to the simulated images based on _B435-dropouts 
in the CDF-S GOODS field as described in Sect. 2.7.1, 
and we recall from Sect. 2.7.2 that these image trans- 
formations have been shown to be representative at the 
> 80% confidence level. 

A mosaic of the CDF-S GOODS tiles is shown in 
Fig. [15] At 2;850< 27.0 mag there are a total of 361 
5475-dropouts in the transformed CDF-S in an area of 
159 arcmin^, giving an average surface density of 2.27 
arcmin"^ (see Fig. [TSt . The surface densities are 1.82 
arcmin^^ and 1.16 arcmin^^ for Z850< 26.5 and Z850< 
26.0 mag, resp. The surface density of (7475-dropouts in 
TN1338 is approximately 2.5 x higher for each magnitude 
cut (5.64 arcmin^^, 4.36 arcmin^^, and 2.74 arcmin^^, 
resp.). 

We d i d not apply the sub-pixel rebinning technique used by 
ILotz et al.l II2006I) that attemps to partially correct for this effect. 



What is the significance of this factor 2.5 sur- 
face overdensity? LBGs belong to a galaxy pop- 
ulatio n that is strongly clustered at ever y red- 
shift (iPorciani fc GiavalScol l2002t lOuchi et al.l l2004bl : 
iLee et al\\2m^ with non-negligible field-to-field varia- 
tions. In our particular case, it is interesting to esti- 
mate the chance of finding a particular number of 5475- 
dropouts in a single 3'A x 3.'4 ACS pointing. Analysing 
each of the 15 GOODS tiles individually, the lowest num- 
ber of (7475-dropouts encountered was 12, and the highest 
was 37 to Z85o=27.0 mag. Next, we measured the number 
of objects in ^ 500 randomly placed, square 11 arcmin^ 
cells in the CDF-S GOODS mosaic. The cells were al- 
lowed to overlap so that the chance of finding the richest 
pointing possible was 100%. In Fig. [16] {top panel) we 
show the histogram of counts-in-cells for the three differ- 
ent magnitude cuts. In each case the number of objects in 
TN1338 (indicated by the dashed lines) falls well beyond 
the high-end tail of the distribution, with none of the 
cells randomly drawn from GOODS containing as many 
objects (the highest being 41, 35, and 24 for Z85o<27.0, 
26.5, 26.0 mag). Approximating the distributions with a 
Gaussian function (strictly speaking, this is only valid in 
the absence of higher order clustering moments, as well 
as non- linear clustering at very small scales), we find a 
surface overdensity of 2.5 at 5 — 6cr significance with re- 
spect to the simulated CDF-S. The measured standard 
deviations were corrected by 4% to take into account that 
the counts in cells distribution will appear narrower due 
to the fact that our control field is not infinitely large. 

Comparable significance for an overdensity is found if 
we focus on a smaller region of 4.4 arcmin^, where more 
than half of the (jf475-dropouts are located (see Fig. [14]). 
Drawing 2.'1 x 2'.1 regions from the CDF-S GOODS field 
(Fig. 1161 bottom panel) yielded a maximum of 21, 19 
and 13 objects for the three magnitude cuts, resp. The 
region in TN1338 corresponds to surface overdensities of 
3.4, 3.2, and 4.0, but with ~ 5a significance due to the 
fact that the counts-in-cells distribution function is much 
wider due to the relatively small cell size compared to 
the fluctuations in the surface density of LBGs (bottom 
panel). 

We have not applied the counts in cells to simula- 
tions of the GOODS HDF-N, since it has been shown 
that the northern GOODS field is ~ 10% less rich 
in i?435-dropouts com pared to its southern counterpart 
(jBouwens et al.l [20071 ). We conclude that the number 
of (7475-dropouts in TN1338 represents a highly signifi- 
cant overdensity with respect to the 314 arcmin^ GOODS 
fields. Below we will further investigate implications of 
large-scale structure and cosmic variance. 

5.3. w{9) and sub-halo clustering 

The two-point angular correlation function (ACF) is one 
of the most powerful tools to study the large-scale distri- 
bution of high redshift galaxies. Recent measurements of 
the clustering of LBGs at 3 < z < 5 show that the ACF, 
w(9), deviates from the classical power-law at small an- 
gular scales. This behaviour is expected in the regime 
where non-linear clustering within single halos dominates 
over the large-scale clustering between halos. This effect 
has now been shown to be pr esent in large LBG samples 
(jOuchi et al.ll2005al: ILee et al. 2006) . 
The structure found in TNI 338 provides a unique op- 
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portunity to test the contribution of the one-halo term to 
■w{6) within a single, overdense field. We measured 'w{9) 
using the estimator w(9) = [AiDD{0) - 2A2DR{e) + 
RR{0)]/RR{0) with A, = NriNr - 1]/{NM „ - 1)) and 
A2 = {Nr - l)/2A^g (|Landv fc Szalavlll993[ ). We used 25 
random catalogs of 10000 sources each, and the errors on 
w{9) were estimated from the standard devia tion among 
32 b ootstrap samples of the original data (jLing et al.l 
[Ml). When the field size is relatively small, the av- 
erage density of a clustered distribution is overestimated 
because the field where the clustering is being measured 
is also the field where the average density has to be 
estimated from. We therefore estimated the integral 
constraint (IC), IC/A^ = / E^ = 

0.073, where we assumed a fixed slope of /3 = 0.6. The 
result is plotted in Fig. [TT] We used bins of 10", but ex- 
cluded the separations of 9 < 1". Since we do not expect 
any signal in the large-scale, 2-halo clustering due to the 
limited size of our sample, we have safely applied the IC 
to the datapoints using the large-scale clustering ampli- 
tude Aw ~ 0.6 for B-dropout s in G OODS at zgso < 26.5 
mag measured bv iLee et al.l (|2006[ ). w{9) is consistent 
with no clustering given the large bootstrap errors. The 
datapoint at 6* < 10" lies above the expected large-scale 
clustering amplitude (solid line). The scale agrees well 
with the expected l ocation of the upturn in w(9) due t o 
sub-halo clustering (jOuchi et al.ll2005aHLee et al.ll2006D . 

To test the significance of possible sub-halo cluster- 
ing in the field of TN1338, we constructed a mock 
field having large-scale clustering properties resembling 
those of LBGs at z ^ 4. We used the formalism of 
ISoneira fc Peeble^ (|1978f ) to create an object distribu- 
tion with a choice two-point ACF. The procedure is as 
follows. First a random position is chosen. This forms 
the center of a pair of points that are placed with a ran- 
dom position angle and separation 9i . Each point forms 
the center for a new pair with separation 02 — 9x/\ and 
random position angles. This process is repeated until 
L levels, each level contributing 2^ points with separa- 
tions 0i/A^^^ to the 'cluster'. Next, a new cluster center 
is randomly chosen in the field, and the cluster is again 
populated with a depth of L levels. This is repeated until 
the mock field contains N clusters. The resulting point 
distribution will have a power-law two-point ACF with 
its slope determined by the choice of A, and its smallest 
and largest angular scales determined by the point sep- 
arations at the first and the last levels, respectively. Be- 
cause we need both many levels to get sufficient signal in 
w{9) at all angular scales, and many clusters to get suffi- 
cient coverage of the area, the method above produces far 
too many points at first. It is therefore common to intro- 
duce a parameter, /, which is the probability that a point 
makes it into the final sample when drawing a random 
subsample. We then calculate the number of clusters N 
necessary to match a particular surface density given this 
/. The amplitude of the ACF, Ayj, solely depends on the 
choice of /, since the clusters are randomly distributed 
with respect to each other. We iteratively created mock 
samples with different / and measured w{9) until the 
best-fit amplitude matched the amplitude of the ACF 
that we wish to model (corrected for the IC). The size of 
the mock field was set to 17' x 17' with a surface density 
of « 5 arcmin"^ to match the density found in TN1338. 



To model the result of lLee et al.l (|2006[ ). w{9) « O.60-°•^ 
we required an / of 0.001 (with 9i = 8f3 and L — 13). 
Having modeled the observed two-point statistics suc- 
cessfully, we extracted 25 3.'4 x 3.'4 'ACS' fields from the 
mock sample and measured the mean w{9) and its stan- 
dard deviation. The result is indicated in Fig. [TTI (shaded 
region). The mean w{9) corresponds to the large-scale 
clustering that was built into the larger mock field (solid 
line). For 9 < 10", the error determined from the mock 
simulations is about half the bootstrap error, and there 
is a 2(7 discrepancy between the clustering observed in 
TN1338 and the expected clustering of a similarly sized 
mock field. We interpret this excess at small scales in 
the TN1338 field as likely being due to sub-clustering of 
galaxies that are physically interacting on scales compa- 
rable or smaller than their typical halo sizes (see Sect. 
ICTl) . 

5.4. Spectroscopy and photometric redshifts 

Excluding the r adio galaxy, 6 of the LAEs confirmed by 
IVenemans e lal. (2002, 2007) are also in our photometri- 
cally selected LBG candidate sample. These large equiv- 
alent width Lya LBGs lie in a narrow redshift interval 
(Az « 0.03) centered on the redshift of the radio galaxy. 
We further obtained spectroscopic redshifts for three of 
the candidate LBGs in our sample. The spectrum of the 

6L* object #367 shows several absorption lines typical 
for LBGs at a redshift of 3.830 ± 0.002 (Fig. \T^. Object 
#3018 was found to have a redshift of z = 3.911 ± 0.004, 
based on the presence of (faint) Lya in emission, and 
Ol/Sill A1303 and GII A1334 in absorption. Candidate 
#959 has strong Lya (as confirmed by its asymmetry) 
at a redshift of z = 3.92 ± 0.01. Although the redshifts 
of these three LBGs in particular indicate no physical 
association with the radio galaxy and Lya emitters, the 
spectroscopic results confirm that our g475-dropout se- 
lection criteria successfully identify LBGs at z « 4. 

We have computed the photometric redshifts of the 
g475-dropouts in TN1338 and the 'simulated' 5475- 
dropouts in GOODS. We let BPZ output the fuU red- 
shift probability distribution for each object, Pi{z), and 
summed over all the objects to get the total redshift prob- 
ability distribution. In this manner, information on the 
likelihoods of all redshifts are being retained, therby im- 
proving the S/N of the total photometric redshift distri- 
bution. The result is shown in Fig. 1181 The area under 
the curves is equal to the total number of objects found 
in a 11.7 arcmin^ area. According to BPZ the fraction 
of galaxies at z ~ 0.5 totals ~ 4% of the candidate z 4 
sample in GOODS. The true contamination fraction is 
likely to be somewhat higher (Giavalisco et al. 2004a). 
The difference in the areas under the two curves refiects 
the factor ~ 2.5 overdensity of the TN1338 field. The 
peak of the zb distribution lies at z — 4.1, which is a 
good match to our target redshift as defined by the radio 
galaxy and the LAEs. The photometric redshift distri- 
bution is steeper around z «:! 4.1 for the (7475-dropouts 
in TN1338 than for those in GOODS. The narrowness 
of the distribution and the overdensity is illustrated by 
subtracting the GOODS zb distribution from that of 
TN1338 (black curve in Fig. [T8|). We conclude that the 
field of TN1338 is overdense compared to GOODS at 
the approximate redshift of z ~ 4.1, but structure mem- 
bership for individual galaxies is difficult to determine 
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given the relative breadth of the redshift distribution of 
S{z) 0.5 (FWHM) based on the present data. We note 
that N(z) has a small secondary peak around z « 3.8. 
Interestingly, its redshift corresponds to the other ~ 6L* 
object in the field (#367 at z = 3.83, Fig. [HI). The 
TN1338 field is exceptional in the sense that it contains 
two ~ 6L* LBGs, while there are only a few of such ob- 
jects in the entire GOODS field. Lya spectroscopy at 
this redshift would be needed to determine whether the 
TN1338 field contains another, overlapping, large-scale 
structure associated with this object as well. 

6. SUMMARY AND DISCUSSION 

6.1. The physical properties of LBGs and LAEs 

SFRs - We studied the star forming properties of 66 
(7475-dropouts to Z850= 27 mag, and 12 LAEs (6 of which 
are also in the g475-dropout sample). The SFRs were in 
the range 1-100 Mq yr"! (Tabled]), with LAEs being 
hmited to < 14 M© yr'^ (Table[2]). Applying an average 
extinction {E{B — V) = 0.1 mag) gives SFRs of up to 
~ 300 Mq yr-i. 

Ages/dust - The LBGs and LAEs have very blue con- 
tinua {Piz « —2) when averaged over the entire sam- 
ple. We derived a UV-slope magnitude relation of (3iz = 
(-0.16 ± 0.05) (z850 - 25 mag) - (1.84 ± 0.11), and found 
that the slopes of the L > L* LBGs are consistent with 
the average slopes determined for mildly reddened LBGs 
at z ~ 3 redshifted to z = 4 (Fig. [§1 iPapovich et al.l 
I2OOII: IShaplev et al]|2003[ ). The - zgso relation can 
be interpreted as a SFR- or mass-extinction relation im- 
plying E{B — V) ~ 0.13 mag at zsso^ 23 mag and 
E{B — V) 0.0 mag at Z850~ 27 mag for a fixed age 
of ~ 70 Myr. 

We derived rest-frame UV-optical colors, and found 
LBG ages in the range 10-100 Myr, with ^ 50% of the 
LBGs having ages < 50 Myr with respect to our base 
template (exponentially declining, r = 10 Myr, Z = 
0.2Z(T) and E{B — V) = 0.16 mag from IPapovich et all 
120011) . We also found evidence for a relation in irr^-Ks 
vs. Ks, similar as found for i?435-dropouts in GOODS 
(Fig. 11 IPapovich e t al. 2004). This is hkely to be in- 
terpreted as a stellar mass-age or mass-dust relation, in 
the sense that more massive galaxies have higher optical 
luminosities and redder UV-optical colors due to aging 
or dust. 

Masses - None of the LAEs was detected in the Ks- 
band, but we found a 3(7 detection through stacking. The 
stacked magnitude of Ks— 25.81q32 mag implied 1775- 
Ks^ 0.0, while a stack of LBGs with similar UV mag- 
nitudes gave a 5a detection of Ks= 25. 14;!^;^^ mag and 
in5~Ks~ 0.7. Although the difference in Ks magnitude 
is only 2(T, other studies based on longer wavelength 
data suggest that LAEs may indeed be fainter at optical 
and in frared wavelengths tha. n LBGs while having similar 
SFRs (jPentericci et al.l[2007t ). The difference can be in- 
terpreted as the LAEs being younger or less massive than 
LBGs. We use the ii's-band magnitudes to derive stellar 
masses, finding ~ 3 x 10* M© for the faintest LAEs and 
LBGs to - 2 X lO^*' Mq for the brightest LBGs (Fig. 
ITU)) . The mean stellar mass of the LAEs in the proto- 
cluster region is in good agreement with the mean stellar 
masses of LAEs selected at z = 3 — 5 (jGawiscr et al. 2006'; 
Pirzkal et afl l2006t iPentericci eFal] I2007i; .Nilsson et all 
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The possible difference bet ween the LAEs and LB Gs is 
qualitatively consistent with lCharlot fc Fall ()1993f ). who 
find that Lya emission may primarily escape during a 
relatively short, dustless phase after the onset of star 
formation. However, observations in Lya and the UV of 
some local starbursts having properties similar to that 
of high redshift LBGs seem to suggest instead that the 
escape of Lya photons may have more to do with the 
properties of the g as kinematics rather than with dust 
(|Kunth et all 120031 ). The nature of the origin of Lya 
emission therefore remains unclear for the moment. 

Sizes - For L > L* galaxies we found a mean 
half-Hght radius of 0'.'22 (~ 2 kpc). Although this 
is slightly sma l ler th an the average size reported by 
'Ferguson et al.' (2004) based on a GOODS B435-dropout 
sample, Ferguson et al,. (2004 ) measured r^i out to much 
larger circular annuli. The results are consistent when we 
apply a small aperture correction from Fig. |3l The mean 
rill of the (7475-dropouts is comparable to that of i?435- 
drop outs culled from the UDF and GOODS fields by 
Bouwens et al.l ()2004aD . and is therefore consistent with 
the oc (1 -|- z)~^-°^^° '^^ size scaling law that connects 
y3nn--B4.s.'^-,Vfinfi-, and i77.r|-dropouts at fixed luminosities 
()Bouwens et al.ll2004aD . The r^ of the L < L* LAEs and 
LBGs are comparable (~ 1.5 kpc). The radii of LAEs in 
the rQ25 filter that contains Lya are similar to those in 
the 4775 filter which images our z ~ 4 sample purely in 
the UV continuum, indicating that the highest surface 
brightness Lya originates from the same region as the 
stellar continuum. 

AGN - Although several of the LAEs appear pointlike, 
the spectra show no evidence for broad Lya or high ion- 
ization lines. Likewise, X-ray observations of a large field 
sample at z ~ 4.5 s how no positive d etecti ons of AGN 
among Lya emitters (jWang et al.ll2004D , and lOuchi et al.l 
(j2007) find an AGN fraction of only ~ 1% among Lya 
emitters at z = 3 — 4. On the other hand, several of 
the LAEs in the protocluster near radio galaxy MRC 
1138-262 at z = 2.16 have been detected with Chandra 
indicating that the AGN fraction of such protoclusters 
could be significant (jPentericci et al . 2002: Croft ct al] 
J005). 

Morphologies - The HST images show that the 5475- 
dropouts span a wide range of morphologies, with some 
showing clear evidence for small-scale interactions (Fig. 
[TT|) . The morphological parameters G, M20 and C de- 
termined for a bright subset of the 3475 dropouts suggest 
that the morphologi es resemble those of i?43R dropou ts 
in GOODS fFig. irai iLotz et al.ll200l iLotz et al.ll200il . 

6.1.1. Summary 

To summarize, the properties of our UV-selected "pro- 
tocluster" sample of LBGs and LAEs are in good agree- 
ment with the physical properties of galaxies in field sam- 
ples. In other words, we find no evidence for trends 
that can be linked to the relatively rich environment 
of the TN1338 structure, in contrast to w hat has been 
observed in lower redshift structur es (e.g. ISteidel et ahl 
120051 iKodama et"ani2007L but see iPeter et all (|2007| )). 
Even if such trends in stellar mass, age, size or morphol- 
ogy exist at z > 4, the relatively crude redshift selection 
applied to select the g475-dropouts at the approximate 
redshift of TN1338 {5{z) - 0.5, FWHM) may wash out 
its signal due to galaxies in the immediate fore- and back- 
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ground. 

6.2. Properties of the protocluster 
6.2.1. Clustering properties 

We have presented evidence for an overdensity of 5475- 
dropouts in TN1338 (Fig. [16]), and that this popula- 
tion has significant sub-clustering across the ACS field 
(Fig. [T4|) . The radio galaxy lies in a ~ 7.5 (co- 
moving) Mpc 'filament' formed by the majority of the 
(7475-dropouts. The discovery of this substructure with 
ACS t ies in closely with the clustering seen at larger 
scales, llntema et al.l ()200 6l) present the large-scale distri- 
bution of relatively bright _B-dropouts towards TN1338 
in a 25' x 25' field observed with the Subaru Telescope, 
showing several significant density enhancements amidst 
large voids. 

In contrast, IVenemans et al.l (|2002l . I2007D . using data 
from two 7' x 7' VLT/FORS fields, found that the LAEs 
are relatively randomly distributed. Based on the ab- 
sense of substructure and their small velocity dispersion 
they concluded that the LAEs might just be breaking 
away from the local Hubble flow. Additional evidence for 
this might be contained in the fact that the LAEs in the 
much smaller ACS field seem to prefer regions that are 
generally devoid of the UV-selected LBGs. We showed 
evidence that LAEs are generally younger (and possibly 
less massive) than objects in our UV-selected sample. 
Also, (?475-dropouts that were detected in Ks are both 
brighter and redder than LAEs, suggesting that objects 
with both higher ages or dust and larger stellar masses 
lie in the densest regi ons of the TN1338 field (see also 
iKashikawa et al.ll2007[ ). This may be expected if the old- 
est and most massive galaxies predominantly end up in 
the population of red sequence galaxies dominating the 
inner part of clusters at z < 1.5. 

We have compared the clustering of the g475-dropouts 
to that expected based on the angular clustering of i?435- 
dropouts at a similar redshift. We found a small excess 
of clustering at the smallest angular scales {9 < 10") 
compared to mock samples with a built-in large-scale 
correlation function (Fig. [T7)l . The small-scale ex- 
cess is to be expected when w{0) is dor ninated by non- 
linea r sub-halo clustering at small scales (iKravtsov et al.l 
12004 lOuchi et al.ll2005at iLee et al.ll2006l ). A radius of 
~ 0.3 — 0.6 (co-moving) Mpc is similar to the virial ra- 
dius, r2n n, of dark inatter h alos with masses of 10^^~^^ 
M0 (see lOuchi et al.ll2005a[ ) . where r2oo is defined as the 
radius of a sphere in which the m ean density is 200 x 
the mean density of the Universe (|Mo fc Whitd I2002D . 
The linear bias, 6, within these radii can reach values of 
> 10 — 50, compar e d to a bias of 2 .5 — 4 for the field 
(jOuchi et al.ll2005at iLee et al.|[2006l ). Detection of this 
small-scale clustering implies that a significant fraction 
of the (7475-dropouts may share the same halo, possibly 
associated with the forming structure near TN1338. 

Given their overall strong clustering properties and 
cosmic variance, z ^ 4 LBGs will likely have significant 
field-to-field variations even on a ngular scales larg er than 
currently probed by GOODS (ISomerville et al.l I2004D . 
Future deep, wide surveys will demonstrate the unique- 
ness of protoclustcr structures such as found in the 
TN1338 field. Based on shallower samples, albeit of sig- 
nificantly larger areas, it has been found that the surface 



density of TN1338-like object concentrations are com- 
parable to that expected based on the co-moving vol- 
umc densi ties of local clusters (e .g. 'Stei del et al.l 119981: 
Shimasaku ^et al.ll2003l : lOuchi et al. 2005bl: llntema et all 
20061) . Observations of these kind of systems could con- 
strain the halo mass function and the halo occupation 
distribution at high redshift at the very high mass end. 

6.2.2. Mass of the overdensity 

A proper determination of the mass of the TN1338 pro- 
tocluster requires a good estimate of the total volume, 
which depends both on the angular and redshift dis- 
tribution of the LBGs and LAEs. We were not able 
to confirm any additional LBGs at the exact redshift 
of the radio galaxy, not only because of the faintness 
and therefore relatively uncertain photometric redshifts 
of the targets, but also because all objects with high 
equivalent width Lya (the most efficient method of con- 
firming objects at these high redshifts) had already been 
found previously in this field. We nevertheless indi- 
rectly obtained redshifts of the 6 .g475-dropouts that are 
also in the Lya sample (excluding the radio galaxy). 
From the redshift distribution of LBGs in the GOODS 
simulations we expect 2.33 field LBGs at a redshift of 
z — 4.1 with \z — Sz\ < 0.03. The volume overden- 
sity in TN1338 is then Sg = {p — p)/p = 1.6, which can 
be considered to be a lower limit since it assumes that 
no other (7475-dropouts lie within 6z of z — A.l. Alter- 
natively, given that only 20%-25% of field LBGs have 
large enough eq uivalent width Lyc^ emission to be de- 
tected as a LAE (Steid el et alll200 tf), the 10 LAEs (with 
Z850< 27 mag) associated with TN1338 would represent 
40-50 LBGs (including LAEs) in a relatively narrow red- 
shift range. Note that if we subtract the average num- 
ber of (?475-dropouts expected in a TN1338-sized field 
(^ 26) from the number of (7475-dropouts observed, the 
surplus amounts to ~ 40 .(7475-dropout s. This is an in- 
dependent c onfirmation of the results of IVenemans et aT] 
(|20f)l [2003) that were based on LAEs alone. 

If we assume that the LBGs in TN1338 have 
the same overd e nsity as me asured for the LAEs by 
IVeneman s et al.| ()2002L 120071 ). the overdensity in the 
TN1338 ACS field would be Sg « 3.5. We can relate the 
true mass overdensity, (5,„, to the observed (7475-dropout 
overdensity, 5g, through 1 -|- bSm — \C\{1 + 6g), where 
C =1 + f - f{l + (5™)i/3 (with / = 17„(z)'*/7) corrects 
for redshift-space distortion due to peculiar velocities as- 
suming that the st ructure is just break ing away from the 
Hubble expansion (ISteidel et al.lfT998h . Taking 6 ~ 3 for 
-B435-dropouts with zs5o< 26 — 27 from lLee et al.l ()2006l ) 
gives Sm ~ 0.8 for 6g = 3.5. This can be related to a to- 
tal mass of M ~ (1 -h Sm)pV > lO" Mq, where p is the 
present-day mean density of the Universe and V is the 
volume probed by our ACS observations. The mass over- 
density corresponds to a linear overdensity of Sl ~ 0.5, 
which when evolved to the present e poch corresponds to 
a hnear overdensity of i5l 2 (see ISteidel et al.l 120051 : 
IOverziei1[20 06b). This exceeds the linear collapse thresh- 
old of 6c = 1.69, so this "protocluster" will have virialized 
by z = 0. We note that these calculations depend on sev- 
eral critical assumptions, such as the si ze of the volume 
(likely to be larger than the ACS field, IVenemans et al.l 
I2OO2I: llntema et al.ll2006f) . the magnitude of the overden- 
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sity (here assumed to be constant over the volume) , and 
the exact bias value corresponding to the objects used to 
determine the overdensity. The validity of these assump- 
tions and the typical properties of such structures will 
be the subject of a detailed comparison with numerical 
simulations in a future work (Overzier et al., in prep.). 

6.2.3. Redshift evolution of the overdensity 

The progenitors of galaxy clusters must have undergone 
rapid and intense star-formation (and possibly AGN ac- 
tivity) at z > 2. The star formation in these 'proto- 
clusters' is not only responsible for the buildup of the 
present-day stellar mass in cluster galaxies, but also for 
the chemical enrichment of the intra cluster medium (e.g. 
lEttoril 120051 IVenemans erall l2005l ). The ages of the 
stellar populations in massive red-sequence galaxies at 
z ^ I are sufficiently high for the m to have begun form- 
ing at rcdshifts 2 < z < 5 (e.g. iBl akeslee et al. l 120031: 
Gjazcbrook ct al. 20 04; TMei et 3.1.1 12006: Homcicr etall 
20061 : [Rettura et aI]l2006L and references therein). 

Simulations suggest that there may be significant dif- 
ferences between the redshift of formation and redshift 
of assem bly for the stel lar mass in massive early-type 
galaxies: iDe Lucia et al.l ([2006) found that for elliptical 
galaxies with stellar mass larger than 10^^ Mq the me- 
dian redshift at which 50% of the stars were formed is 
~ 2.5, but the median redshift when those stars were ac- 
tually assembled into a single galaxy lies only at ^ 0.8. 
In the same paper, they showed that the star formation 
properties of ellipticals depend strongly on the environ- 
ment. For elliptical galaxies in clusters, the average ages 
can be up to 2 Gyr higher than those of similar mass 
ellipticals in the field. For an elliptical that is ^ 1 Gyr 
older compared to the field, 50% of its stellar mass will 
already have been formed at z ~ 4. This stellar mass 
is likely to be formed in much smaller units, while the 
number of major mergers is considered to be relatively 
small (a few). 

It is likely that the stellar mass formed by protocluster 
galaxies will end up in quiescent cluster early-types at 
lower redshifts, in accordance with the color-magnitude 
and morphology-density relations. However, there is a 
significant discrepancy between the masses of the LBGs 
and LAEs (both in protoclusters and in the field) and 
the masses of cluster early- type galaxies of > 10, indi- 
cating that a large fraction of the stellar mass still has 
to accumulate through merging. Detailed observations 
of protocluster regions on much larger scales (~50 co- 
moving Mpc) are needed to test if the number density of 
LBGs is indeed consistent with forming the cluster red 
sequence population through merging. 

Alternatively, protocluster fields may also host older 
galaxies of significant mass, analogous to the popula- 
tion of distant red galaxies found at 2 < z < A (e.g. 
iFranx et all 120031: Ivan Dokkum et all 120031: IWebb et al.l 



l2006f ). These objects are believed to be the aged and red- 
dened descendants of LBGs that were UV luminous only 
at z > 5 — 6, and form a population that is highly clus- 
tered (|Quadri et al.ll2007l ). Attemps are currently being 
made in finding such objects toward prot oclusters (e.g. 
iKajisawa et al.ll2006l : iKodama et al.ll2007t l. but spectro- 
scopic confirmation is difficult. 

6.3. Comparison to simulations 

Our results are in agreement with studies of large-scale 
structure and pro toclusters using A^-body simulations. 
ISuwa et all (l2006f ) studied global properties of protoclus- 
ters by picking up the dark matter particles belonging to 
clusters at z = and tracing them back to high red- 
shift. The simulations showed that clusters with masses 
of > 10^* Mq can be traced back to regions at z = 4 — 5 
of 20-40 Mpc in size, and that these regions are asso- 
ciated with overdensities of typical halos hosting LAEs 
and LBGs of i5g '--^ 3 and mass overdensities 6m in the 
range 0.2-0.6. For randomly selected regions of the same 
size, the galaxy and mass overdensities were found to be 
mostly < 0, as expected due to the fact that massive ha- 
los are relatively rare. Although some of the overdense 
regions in the simulations having a similar overdensity as 
our protocluster candidate do not end up in clusters at 
z = 0, the simulations show that most regions with an 
overdensity on the order of a few at z 5 will evolve into 
clusters more massive than 10^'* M© {> 50% for Sg >2). 

Also, recent numerical simulations of CDM growth 
predict that quasars at z ^ 6 may lie in the cen- 
ter o f very ma ssive dark matter halos of ~ 4 x 10^^ 
Mq (jSpringel e t al. 2005t iLi et aT1l2006l ). They are sur- 
rounded by many fainter galaxies, that will evolve into 
massive clusters of ~ 4 x 10^^ Mq at z = 0. The discov- 
ery of galaxy clustering associated with luminous radio 
galaxies and quasars a t z > 2 (e . g . 'Stiav elli et al.l 120051 : 
IVe nemans et al.l 120071 : IZheng et al., 2006. this paper) is 
consistent with that scenario. 
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TABLE 1 
Summary of observations. 



Filter 


Date 


Texp 


A 


Depth 






(s) 


(mag) 


(2a) (5a) 


9475 (F475W) 


2002 July 11-12 


9400 


0.359 


28.46" 27.47° 


r625 (F625W) 


2002 July 8-9 


9400 


0.256 


28.23" 27.23" 


1775 (F775W) 


2002 July 8-9 


11700 


0.193 


28.07" 27.08" 


Z850 (F850LP) 


2003 July 11-12 


11800 


0.141 


27.73" 26.73" 


Ks 


2002 March 24-26, 2004 May- July 


27000 


0.036 


25.15' 24.16'' 



^ Measured in 0''45 diameter square apertures. Measured in 1''4 diameter circular apertures. 



TABLE 2 

Properties of the spectroscopically confirmed Lyo emitters. 



ID 


O:j2000 


l5j2000 


Zspec 


(9475-^625)" 


(r626-Z860)° 


(1775-2850)" 


2850 




rhl,z 


SFR'fjy 


RG 


13:38:26.05 


-19:42:30.47 


4.105 


3.42 ±0.17 


-0.61 ±0.03 


0.09 ±0.03 


23.05 ±0.05 


O'.'OO 


0'.'62 


qq 7+2-71 
' -2.63 


L4 


13:38:22.46 


-19:44:33.67 


4.095 


1.88 ±0.27 


-0.35 ±0.15 


-0.14 ±0.16 


26.68 ± 0.23 


0'.'09 


0('08 


4.83l°;|! 


L7 


13:38:24.78 


-19:41:33.66 


4.106 


> 1.26 


0.65 ±0.29 


0.32 ± 0.25 


27.20 ± 0.49 


0'.'18 


O^'ll 


•^•^^-0.67 


L8 


13:38:24.86 


-19:41:45.49 


4.102 


> 1.47 


0.37 ±0.27 


-0.26 ±0.23 


26.51 ±0.30 


0^12 


0^'16 


6 25+°-** 


L9 


13:38:25.10 


-19:43:10.77 


4.100 


1.78 ±0.25 


0.49 ± 0.09 


-0.02 ±0.07 


25.34 ±0.08 


0('12 


0^14 




Lll 


13:38:26.16 


-19:43:34.31 


4.101 


1.56 ±0.18 


0.11 ±0.09 


-0.08 ±0.09 


25.94 ±0.10 


0'.'08 


0'.'09 


8.95t°;S 


L14 


13:38:28.72 


-19:44:36.98 


4.102 


> 1.88 


0.37 ±0.18 


0.24 ±0.18 


26.52 ± 0.18 


0'.'13 


0^'13 


4 2q+°-5'' 
'*-^y_0.48 


L16 


13:38:29.66 


-19:43:59.82 


4.102 


1.43 ± 0.23 


0.22 ±0.12 


-0.00 ±0.11 


25.54 ±0.16 


0^'16 


0^'19 


11 4+^-1° 

^^•^-1.00 


L17 


13:38:29.86 


-19:43:25.84 


4.093 


> 1.57 


0.21 ±0.27 


0.44 ±0.29 


27.37 ±0.28 


0^11 


O'.'IO 


1 qn+0.40 

^■■'•^-0.31 
r; 1 0+0.53 
"'■-^^-0.48 


L20 


13:38:32.83 


-19:44:6.934 


4.100 


> 2.17 


0.37 ±0.15 


0.13 ±0.14 


26.44 ±0.16 


0('09 


O'.'ll 


L21 


13:38:33.56 


-19:43:36.00 


4.097 


1.50 ±0.32 


-0.00 ±0.18 


-0.16 ±0.18 


26.22 ±0.19 


0^'12 


0"18 


4 75+0.79 


L22 


13:38:34.14 


-19:42:52.68 


4.096 


> 1.82 


0.79 ±0.16 


0.13 ±0.13 


26.64 ± 0.14 


0'.'07 


0('08 


4.69l°;i 


L25 


13:38:34.96 


-19:42:24.95 


4.093 


1.68 ±0.33 


0.32 ±0.13 


0.00 ± 0.12 


25.81 ± 0.15 


0^25 


0'.'24 


9.941^7^ 



^ Isophotal colors. The limits are 2a. Total magnitudes.*^ SFR estimated from the UV continuum flux (2775). 
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TABLE 3 

Properties of the 2: ~ 4 Lyman break sample. 
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Fig. 1. — Total e ffective throug hput of the HST/ACS g475»'625i775^850 a-nd VLT/ISAAC Ks filters. The galaxy spectru m shown is the 
SB2 template from [Bem'tezi II200Q1 redshifted to 2 = 4.1, and applying the attenuation prescription of lMadau et al.l II1996I) . The GOODS 
-B435 and Veoe filters are indicated by dashed lines for comparison. 




Fig. 2. — The ACS field showing 3475 in blue, rg25 in green and ^gso in red. The field measures 11.7 arcmin^. The radio galaxy TN1338 
stands out clearly as the bright green object (white circle). 
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Fig. 3. — Left: Intrinsic rf^i ^ versus r^j ^ measured by SExtractor for de Vaucouleurs profiles (thick lines) and exponentials (thin lines). 
The difference between intrinsic and measured radius is smaller for exponentials. The intrinsic sizes are increasingly underestimated when 
going to fainter magnitudes, e.g. from 2g5o~ 24 mag (top lines) to zgso"^ 26 mag (bottom lines). — Middle: The difference in zgso between 
MAG_AUTO and total 'intrinsic' magnitudes for de Vaucouleurs profiles (thick line) and exponentials (thin line). Right: Completeness 
limits in zgso as a function of total 'intrinsic' magnitudes and rjjj, where completeness is defined as the ratio of the number of objects 
detected to the number of artificial objects added to the image (50% exponential; 50% de Vaucouleurs). The 50% completeness limit lies at 
an intrinsic magnitude of fs; 26.5 for small sources. When expressed in terms of MAG_AUTO, the completeness limits shown here should 
be some ~0.5— 1.0 mag fainter, due to an underestimate of the 'total' flux for faint galaxies. 




Fig. 4. — 9475 -'"625 versus ?'625— zgso for model simulations using the Bruzual & Chariot' (200?) libraries. Grey points indicate the modeled 
colors of a large set of SEDs in the redshift range < z < 6 (see text for details). Galaxies at 2; = 4.1 with ages less than 500 Myr are shown 
as large solid circles. Our selection criteria for selecting (;475-dropouts at z ^ 4.1 as defined by 5475-?'625> 1.5, g475-r'625 >^'625^285o+l.l, 
''625^2850 5: 1-0 a-re indicated by the shaded area. The spectral tracks are an elliptical (red solid line), an Sbc (red dashed line), an Scd 
(red dotted line), and a 100 Myr constant star formation model with E{B — V) = 0.0 mag (blue sol id line) and E( B — V) = 0.2 mag (blue 
dotted line). Redshifts are indicated along the tracks. The redshift of the overdensity o f Yencm anset al.l 120021) is marked by the large 
blue squares (2 4.1). Stars (green) mark the stellar locus based on the stellar library of IPickle3 ||1998I) . 
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Fig. 5. — Selection efficiency for 2 ~ 4 LBGs. The dotted histogram shows the fraction of model galaxies that meets the selection 
criteria in each redshift bin. The solid histogram shows the selection efficiency for model galaxies with ages less than 100 Myr and 
< E{B — V) < 0.3. The dashed histogram shows the fraction of models with ages greater than 0.5 Gyr selected, illustrating possible 
contamination of our z ~ 4.1 sample by relatively old galaxies at ^ ~ 2.5. A (minor) source of contamination is the possible inclusion of 
Balmer-break objects at 2: ~ 0.5. The shaded region indic ates the redshift interval (2 = 4.07 — 4.13) of the protocluster defined by the radio 
galaxy and the hya emitters from I Venemans et al.l 1)20021 ) . 
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Fig. 6. — Colors and (;475-dropout samples derived from two different simulations of the TN1338 (?475f'g25*775Z850 data set. In the 
first simulation (left panel), the input model was transformed directly in terms of the TN1338 S475T"625*775-285ofilter set. In the second 
{right panel), the model was first transformed in the terms of the GOODS -B43 5 V"60 6^775 zsspfilter set and then transformed to the TN1338 
5475^625*775-2850 filter set using the transformation method laid out in Section 12.71 Objects detected in the images are indicated by small 
circles. Objects falling within our (;475-dropout selection window (marked by thick lines) are indicated by the large squares. Note that 
the transformation from model to GOODS to TN1338 {right panel) introduces some small changes in the color distributions with respect 
to the transformation directly from model to TN1338 {left panel). This is due to the fact that in the right panel object detection and 
color selection are performed twice, and because of uncertainties introduced by determining the photometric redshifts of the objects in the 
simulated GOODS images. However, the 3475-dropout selections obtained from the images generated from our transformation method are 
nearly identical in number to those obtained from direct simulations of the same data set. This suggests that it is reasonable to use the 
(?475-dropout samples obtained from the transformed images for quantifying the expected number densities (see Section |5j. 
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Fig. 7. — Color-color diagram of (;475-dropouts in TN13 38 (large circles) and t he detection catalog (grey points). The shaded region 
shows our selection window (Eq. 1). Confirmed LAEs from FVenemans et al.l II2002I ) are marked by red stars, a nd the radio galaxy by the 
large black square. Small green stars mark the stellar locus based on the stellar SED library of IPickle^ II1998I) . See the caption of Fig. |4] 
for further details. 
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23 24 25 26 27 

■^850 

Fig. 8. — zg^o versus ft^ for g475-dropouts detected/undetected in Ks (filled/open circles), LAEs (stars), and the radio galaxy (square). 
The best-fit linear relations are indicated (thick lines, see text for details ) . Th e thin solid line is the relation for ,8435 -dropouts (R.J. 
Bouwens, private communication). The best-fit SED from iPapovich et al.l 11200 Ih redshifted to 2; = 4 has fSi^ fa —1.4 (thin dashed line). 
Shaded regions are for E{B — V) = 0.0 mag with ages between 0.05 and 0.3 Gyr (bottom light shaded region), E{B — V) = 0.1 mag with 
ages between 0.005 and 0.15 Gyr (dark shaded region), and E{B — V) = 0.3 mag with ages between 0.001 and 0.3 Gyr (top light shaded 
region), assuming an exponential star formation history (t = 10 Myr) with 0.2^0 metallicity and a Salpeter IMF. 
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Fig. 9. — Rest-frame UV-optical colors of the g475-dropouts (circles), the LAEs L9 and L25 (stars), and the radio galaxy (square). 
Arrows indicate 2u limits for non-detections in Ks (errors omitted for clarity). Lines indicate the colors of a r = 10 Myr SED (0.2Zq) 
with ages in Gyr along the track for E(B — y) = 0.0 mag (dotted) and E{B — V) = 0.15 mag (solid). The large circle was obtained from 
a ii's-band stack of 12 51475 -dropouts having 25.3<i775 <26.4 mag. The large star was obtained from a /sTg-band stack of 5 LAEs within a 
similar magnitude range. Their 177^— Kg colors differ by ~ 0.7 mag. 
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Fig. 10. — Color-magnitude diagram of the (;475-dropouts (circles). The dashed line indicates the approximate 2(t detection limits. The 
tracks are for t = 10 Myr SEDs with different stellar masses of 0.03, 0.1, 0.5 a nd 2x10^° Mfn for E{B ~ V) = 0.0 mag (dotted) and 
E{B — V) = 0.15 mag (solid). The thick solid line indicates the 'blue envelope' of IPa povich et aLl l[2004), and suggests a color-magnitude 
relation in which luminosity correlates with either age or dust. See the caption of Fig. [9] for further details. 




Fig. 11. — 1775 postage stamps (3" x 3") of the p476-dropout sample. The images have been smoothed using a Gaussian kernel of (/.'OTS 
(FWHM). Kron apertures are indicated. The scale bars measure 1". North is up, East is to the left. 
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Fig. 12. — 1775 postage stamps (3" X 3") of the twelve spectroscopically confirmed Lyo emitters of IVenemans et al.l II2002I) . See the 
caption of Fig. Illl for details. 
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Fig. 13. — The morphological Gini coefficient (G), M20, and concentration (C) for the g475-dropouts in TN1338 (blue points and 
contours). The parameter distributions determined from ~70 LBGs at 2; ~ 4 selected in the GOODS CDF-S field are shown for comparison 
(red contours). The radio galaxy is indicated by the large black circle. The approximate size of the error bars on the measurements are 
indicated in a corner of each panel. 




Fig. 14. — Object map of the (;475-dropout candidates (circles), TN J1338-1942 (square), Lya emitters (stars) and the detection catalog 
(points). (;475-dropouts detected in Ks are indicated by filled circles. Larger circles indicate brighter objects in 2:850. The contours represent 
density fluctuations A = (S — S)/S of -1, —0.5, -0.1 (dashed contours) and +0.1, +0.5, +1, +1.5 (solid contours), achieved by smoothing 
the object map with a Gaussian of width 36", or 250 kpc (FWHM), using equal weights. LAEs that are not in the g475-dropout sample 
were not included in the density contours. 




Fig. 15. — Distribution of z ~ 4 LBGs in the CDF-S GOODS field with Z8bo< 27.0 mag (circles). Larger symbols correspond to brighter 
objects. The inset in the top left shows the 3.'4 x 3.'4 TN1338 field and the distribution of g47B-dropouts at the same scale as the CDF-S 
GOODS field for comparison. 
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Fig. 16. — Counts in cells analysis of 2 ~ 4 LBGs in the GOODS simulations compared to TN1338. Top panel: Histograms of the 
number of objects in square cells the size of TN1338 (3.'4 X 3'.4) for 2g5o<27.0 (blue), 26.5 (green) and 26.0 mag (red). The number of 
(?475-dropouts in TN1338 are indicated by the vertical lines of corresponding color. Bottom panel: Same as top, but for 2.'1 X 2.'1 cells. 
The number of (;475-dropouts in TN1338 exceeds the number encountered in GOODS for each limiting magnitude and cell size. 
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Fig. 17. — The angular two-point correlation function for 3475-dropouts in the TN1338 field (solid points). The solid line indicates 
the angular correlation function w(d) = 0.66""'®, as measured for iJ435-dropouts with zgso^ 26.5 mag in GOODS (Lee et al. 2006). The 
shaded region indicates the la spread in w{6) at each 9 that was measured among 25 3f4 X 3.'4 fields of the same geometry and source 
density as the TN1338 field extracted from a 17' X 17' mock catalog with w{e) « 0.69^°-^. 
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Fig. 18.— Total zb probability distributions for 9475-dropouts in TN1338 (blue) and in the CDF-S GOODS field (red). The area under 
the blue curve is equal to the number of objects in the TN1338 sample. The red curve for GOODS has been normalised to the area of 
the TN1338 field. The GOODS curve was subtracted from the TN1338 curve to bring out the residuals of the redshift distribution for 
TN1338 (black curve). The residual peaks at the redshift of the radio galaxy and LAEs (shaded region). A secondary peak of 1 object lies 
at zg ft! 3.8, the redshift of the other ~ 6L* object in the field. The peaks at 2; ~ 0.5 are due to the alternative probability that a fraction 
of the (;475-dropouts could be 4000A break objects. 
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Fig. 19.— VLT/FORS2 spectrum of object #367 with a redshift of 3.830 ± 0.002. 



